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(54) Image Processing System using image ciemosaicing 



(57) A method for operating a data processing sys- 
tem to generate a full color image from a partially sam- 
pled version of the image The full color image includes 
a first two-dimensional array of vectors having compo- 
nents representing the intensity of a pixel in the full coior 
image in a corresponding spectra) band at a location de- 
termined by the location of the vector in the first two- 
dimensional array. The method generates the first two- 
dimensional array from a two-d?mensional array of sca- 
lars. Each scalar determines one of the first, second, or 
third intensify values at a corresponding location in the 
two-dimensional array of vectors. The method deter- 
mines the remaining ones of the first, second, and third 
intensity values. The method starts by assigning a value 
to each one of the components of the vectors in the first 
two-dimensional array of vectors that is not determined 
by one of the scalars. A luminance image and first and 
second chrominance images are then generated from 
the first two-dimensional array of vectors. The chromi- 
nance images are filtered with an isotropic low-pass 
spatial filter to generate tittered chrominance images. 
The two-dimensional array of vectors is then regenerat- 
ed from the luminance image and the first and second 
filtered chrominance images. The scalars that are orig- 
inally given by the sensors are reset in the regenerated 
two-dimensional array of vectors. The decomposition, 
filtering, resetting, and regenerating steps are iterated 
to provide the final full-color image. In the preferred em- 
bodiment, the luminance image is also filtered. Howev- 
er the filtering ot the luminance image utilizes a low- 
pass spatial filter having an anisotropy that varies with 
location in the luminance image. 
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Description 

[0001 ] The present invention relates to a method of operating a data processing system and to a method of interpo- 
lating an image for use, for example in digital cameras and similar devices, for converting data from a camera sensor 
s to a color image. 

[0002] A digital cofor image usually consists of an array of pixei values representing the intensity of the image at 
each point on a regular grid. Typicaliy, three colors are used to generate the image. At each point on the grid the 
intensity of each of these colors as specified, thereby specifying both the intensity and color of ihe image at thai grid point 
[0003] Conventional color photography records the relevant image data by utilizing three overlapping color sensing 
to layers having sensitivities in different regions of the spectrum (usually red, green, and blue). Digital cameras and 
scanners, in contrast, typically utilize one array of sensors in a single "layer", 

[0004} When only one sensor array is used to detect coior images, only one color may be detected at any given 
sensor location. As a result, these sensors do not produce a color image in the traditional sense, but rather a collection 
of individual color samples, which depend upon the assignment of color filters to individual sensors. This assignment 
is is referred to as the cotor fHter array (CFA) or the color mosaic pattern. To produce a true color image, with a full set 
of color samples (usually red, green and blue) at each sampling location, a substantial amount of computation is re- 
quired to estimate the missing information, since only a single color was originally sensed at each location in the array. 
This operation is typically referred to as "demosa icing*. 

[0005] To generate the missing information, information from neighboring pixels in the image sensor must be used. 
20 in addition, some assumption must be utilized about the structure of the underlying irnage ? since there are an infinite 
number of possible images that could have generated the measured color values. Typically, it is assumed that the 
underlying image is smooth, and an interpolation algorithm is then utilized to compute the missing color vaiues from 
the neighboring measured color values. 

[0006] While most images of interest to human observers are mainly smooth, the smoothness assumption is not 
2& satisfied along edges of objects and in textured regions of the image. In these regions, images generated by interpo- 
lation using the smoothness assumption show a toss of resolution. In addition, algorithms that treat the red sensors 
as being independent from the green sensors, and soon, typically generate color artifacts in the reconstructed images. 
The artifacts are incorporated in the chrominance part of the image and are due to misregistration of the chrominance 
components. These artifacts express themselves as streaks of false colors in the restored image , and are especially 
30 apparent around boundaries between different objects in the image and in textured areas. 
[0007] The present invention seeks to provide improved image processing. 

[0008] According to an aspect of the present invention there is provided a method of operating a data processing 
system as specified in claim l. 

[0009] According to another aspect of the present invention there Is provided a method of interpolating an image as 
35 specified in claim 6. 

[0010] The preferred embodiment can provide an improved image processing method for converting dala from a 
pixel array having non-overlapping sensors to a fully sampled digital image. 

[0011] It is also possible with the preferred embodiment not to generate the cofor artifacts discussed above and/or 
which can have improved resolution around boundaries between objects and in textured areas. 

40 [0012] The preferred embodiment provides a method of operating a data processing system to generate a full color 
image from a partially sampled version of the ^mage. The full color image concludes a first two-dimensional array of 
vectors. Each vector in the first two-dimensional array includes first, second, and third intensity values. Each intensity 
value represents the intensity of a pixel in the full color image m a corresponding spectral band at a location in the full 
color image determined by the location of the vector in the first two-dimensional array. The method generates the first 

45 two-dimensional array from a two-dimensional array of scaurs., each scalar corresponding to one of the vectors in the 
first two-dimensional array The scalar determines one of the first, second : or third intensity values at a corresponding 
location in the two-dimensional array of vectors. The scalars are typically the measured pixei values from a sensor 
arras/ in a digital camera or the like. The preferred method determines the remaining ones of the first, second, and third 
intensity vaiues. The metnod starts by assigning a value to each one of the components of the vectors in the first two- 

&o dimensional array of vectors that s not determined by one of the scalars. A luminance ?mage and first and second 
chrominance images are then gen&mXed from the first two<Jimensionai array of vectors Each pixe^ in the luminance 
image ^s determined by the intensity of iight from a corresponding pixel in the first two-dimensional array, and each 
pixel in the first and second chrominance images representing the color of the pixei in the first two-dimensional array. 
The first and second chrominance images are filtered with an isotropic low-pass spatial filter to generate first and 

ss second filtered chrominance images. The two-dimensional array of vectors is then regenerated from the luminance 
image and the first and second filtered chrominance images. The decomposition, filtering, and regenerating steps are 
iterated to provide the final full-color image. At each iteration, the components of the two-dimensional array of vectors 
that were actuary measured are reset to their measured vaiues. m the preferred embodiment of the present invention, 
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the luminance image is also filtered. However, the filtering of the luminance image utilizes a low-pass spatial fitter 
havsng an anisotropy that varies with location m the luminance image. The regeneration step is then performed with 
the filtered luminance image. In the preferred embodiment of the present invention, the filtering of the luminance image 
is performed by first decomposing the luminance image Into a plurality of component images each component image 
representing Information m the luminance image at different levels of scale. Each component image is then fHtered 
with a iow-pass spatial filter having an anisotropy that varies with location in the component image. The filtered com- 
ponent images are then recombined to generate the filtered luminance image. 

[0013} An embodiment of the present invention is described below, by way of example only with reference to the 
accompanying drawings, in which: 

[0014] Figure 1 illustrates the demosaicing problem as applied to a typical image taken with an image sensor having 
a repeating 2x2 pattern. 

[001 5] Figure 2 is a flow chart of a preferred method for filing in the missing pixel values in an image having a single 
color band. 

[0O1 6] Figure 3 is a ffow chart of the preferred embodiment of demosaicing method. 

[0017] The preferred method, disclosed herein, may be applied to any color-sampling device: however to simplify 
the following discussion, the method will be explained in terms of an image sensor having a sensor pattern that is 
constructed by repeating a kernel of sensing elements For example, one common image sensor array is based on 
the Sayer pattern v^h\ch is generated by repeating a 2x2 sensor array kernel havmg two green sensors, one red sensor, 
and one blue sensor. This pattern is shown in Figure 1 at 10. The kernel is shown at 12. 

[0018] The method converts the data from such an array into an augmented array shown at 10' in which each pixel 
has intensity values representing each of three colors. The missing colors at each location are determined from the 
nearby sensor values for that cofor. The missing colors are shown in a smaller typeface in the figure The individual 
sensor locations are assigned an address of the form (am), where n is the row number and m is the column number 
in which the sensor is located. 

[001 9] There are an infinite number of patterns 1 0' that satisfy the constraint that the measured values are as shown 
in 1 0- Hence, some additional constraints must be applied in generating the sensor values. The method assumes that 
the image is smooth except in the regions dominated by edges or texture. Consider an image having an object therein 
with a sharp edge. The object is assumed to be smooth near the edge. That is, if one moves through the pixels in a 
direction parallel to the edge ; the sequence of intensity values changes smoothly. In contrast if one moves through 
the pixel values in a direction perpendicular to the edge, the sequence of intensity values changes abruptly at the edge. 
[0020] As stated above, there are two main problems in the reconstructed image: The color artifacts along the edges 
and loss of resolution due to edge smoothing. The manner in which the preferred method deals with the loss of resolution 
due to edge smoothing will be explained first. Consider a mono-chromatic image having missing pixel values in which 
the missing pixel intensities are to be computed. This problem is essentially the same as the problem of computing 
the missing cofor pixels for one color without considering the pixel values from the other co^or sensors. The method 
assumes that the measured pixel values are not to be changed, in addition, the image is assumed to be piecewise 
smooth with sharp edges. 

[0021] This simplified version operates as follows. First a guess is provided for each of the missing pixels. For 
example, the missing pixel values can be initially set by interpolating the nearest four known pixels around the pixel in 
question. The specific method for providing the starting estimate is not critical: hence, any convenient method may be 
utilized. 

[0022] Next the pixels are processed one at a time as follows. First, the direction and amplitude of maximum descent 
in the image at the pixel being processed is determined. The simplest algorithm for making this determination utilizes 
a weighted sum of the differences in pixef intensity between the pixel being processed and its neighbors: however 
more complex methods may be utilized if there is an edge at the pixel in question, the edge wHi be perpendicular to 
trie direction of maximum descent The amplitude of the maximum descent determines whether an edge passes through 
this pixel. Hence, a 2D vector is defined for each pixel; its direction provides the edge orientation and its length the 
edge strength. This vector is referred to as *the dominant orientation vector 14 in the following discussion. The pixel in 
question is interpolated by computing a weighted sum of values of neighboring pixels. The weights of this sum are 
determined by the dominant orientation vector associated with this pixel. If the length of the dominant orientation vector 
is h^gh. only pixel values along the dominant orientation are summed. H the length of the dominant orientation is tow 
or zero (i.e,,na edge) neighboring pixel values from all adjacent pixels are summed. 

[0023] After ail of the image points have been processed, the pixels corresponding to the measured pixeis are reset 
to the measured values. The process is then repeated until the difference in the successive images is below some 
predetermined error value. 

[0024] The number of points that are used in determining the direction of maximum descent and in smoothing the 
image ideally depend on the scale of the objects m the image. For exampre ; if the distance over which points are 
selected for the smoothing operation is loo smalt then insufficient smoothing will occur and the resultant image will 
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have artifacts near the edges. 11 the distance is too large, structural details in the objects will be blurred. The simplified 
procedure described above does not measure the scale of the objects in the image, and hence, cannot provide an 
optimal distance, which, in general will change with the point being processed. Therefore, it is preferable to appfy the 
above process in a mufti-scale manner but a single scale procedure as described above is also possible 

$ [0025] Refer now to Figure 2 which is a block diagram of a preferred method for filling in the missing pixel values in 
an image having a single color band. The original image is first interpolated to provide starting values for the missing 
pixels as shown at 22- In the preferred embodiment of the present invention, the scale ot the objects in the image is 
taken into consideration by first decomposing the image into a number of "component" images that represent the image 
at various levels of scale as shown at 23. The smoothing operation described above is then applied to each of the 

io component images as shown at 24. The component images are then recombined to recover a new image as shown 
at 25. The pixels of the new image at the locations at which intensity values were actually measured are then reset to 
the measured values as shown at 26. The process is then iterated as shown at 27 until the difference in successive 
recovered images is below some predetermined error value. 

[0026] The preferred method tor decomposing the image into a plurality of component images containing details of 
1& differing scale is to generate a Laplacian pyramid The decomposition of an image using this approach is well known 
in the image processing arts, and hence, will not be discussed in detail here. The reader is referred to R J. Burt and E. 
H. Adelsom ""The laplacian pyramid as ^compact image code®, IEEE Trans. Communications, 31{4):S32-540 : 1983 
for a detailed discussion of the method For the purposes of this discussion, it is suffiaent to note that a sequence of 
images is obtained from the original image as follows. The original image is first blurred by applying a Guassian biurring 
20 operator to the image. The blurred image is then subtracted from the original image to form a component image having 
details at the highest spatial frequencies } re, 5 the smallest scale. The blurred mnage is then aown-sarnpled, and the 
process repeated to generate a component image at the next larger scale by repeating the above-described process 
with the down-sampled image in place of the original image. The various component images can be combined to 
regenerate the original image. 

2S [0027] The smoothing operation is applied separately at each level in the pyramid. For each pixel in a component 
image, a direction is defined as described above. The pixel being processed is then replaced by the result of smoothing 
the neighboring pixels along that direction. 

10028] After all of the component images have been smoothed, the component images are recombined to generate 
a new image. The pixel values at the locations at which the original image was actually measured are then reset to 

so the measured values and the process repeated until a predetermined degree of convergence has been obtained. 

[0029] The preferred method for defining the preferred orientation for the smoothing operator is steerable filters. The 
use ot such filters to determine the local orientation of the image is well known m the image processrng arts, and hence, 
will not be discussed in detail here. The reader is referred to W.T Freedman E H. Adeison, "Steerable filters for early 
vision, image analysis and wavelet decomposition*, in International Gonf. on Computer Vision, 1990 7 pp. 406-415. For 

■35 the purposes of this discussion, it is sufficient to note that a set of filters can be defined that measure the "directional 
energy" in the image in a predetermined direction. By combining the output of the filters, the direction having the 
maximum energy can be determined. The maximum energy determines the "dominant orientation" vector whose length 
and orientation are determined according to The maximum energy value and its direction. 

[0030] Denote the dominant orientation for pixel by the vector D{i.\ }. The interpolated value l{i,j) for pixel (kj) is 
40 obtained computing a weighted sum of pixels l(k,i) in a local neighborhood, N(i J), of pixel The weights of the sum 
are determined by the dominant orientation D(i,j). Prxels whose direction from (ij) are parallel to D(iJ) are weighted 
higher than pixels perpendicular to D(i,j) in the preferred embodiment of the present invention, 
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where the sum is over aH pixels (k.i) in N(ij). Here, the weights, W(kl), are defined by: 

WikJyn-Diijl-ViJjX!)) 



where D*V is an inner product and Vfi.j.kJ) is defmed as the unit vector pointing from (ij) towards fkJ). 
ss [0031] The above-described embodiments operate on a single cofor band. In pdnciple, the demosa icing of a color 
image can be performed by applying the directional smoothing described above to each color band {red, green, and 
blue) separately. However such an approach leads to color artifacts, particularly at the edges of objects, due to mis- 
registration in the color bands. 
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[0032] The color artifacts can be reduced by performing the smoothing computation m the YIQ rather than the RGB 
representation of the image. In the YIQ representation the Y component represents the luminance (intensity) of the 
image and the i and Q components represent the chrominance. The luminance-chrominance representation is an 
uncorrelated color representation, and hence, each color band can be treated separately. Further, the human visual 
system is more sensitive to high spatial frequencies in the luminance band (Y) than in the chrominance bands (IQ). In 
fact, in the chrominance bands, the human visual system is only sensitive to low spatial frequencies. 
[0033] The YIQ representation can be computed from the RGB representation via a simple linear transformation: 
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[0034] Since edges, lines, and textures are composed of high frequencies, \hmr appearance is influenced mainly by 
the luminance component. Accordingly, an isotropic smoothing can be provided in the chrominance bands while utilizing 
a directional smoothing in the luminance band. The isotropic smoothing substantially reduces the color artifacts while 
20 the directional smoothing preserves high frequency structures such as edges. In the preferred embodiment of the 
present invention, the isotropic smoothing is provided by smoothing ! and Q bands by convolution with a Gaussian 
kernel. The same kernel used to generate the Lapiacian pyramid may be utilized here. 

[0035] Accordingly, the preferred embodiment of the present invention for demosaicing a color image includes two 
pathways as shown in Figure 3 which is a block diagram of the preferred embodiment of demosaicing method. The 

& original image is first interpolated to provide starting values for the missing pixel intensities as shown at 31 , The image 
is then separated into a luminance band, Y and two chrominance bands, I and Q as shown at 38 The luminance band 
is directionally smoothed using'the method described above or any directional smoothing known in the literature. The 
chrominance bands are isotropicaily smoothed by a convolution with a Gaussian kernel as shown at 38. The resulting 
values are then transformed back into the RGB representation as shown at 39. The pixel values at those locations that 

^0 were actually measured m each band are then reset to their original values as shown at 36. The process is then iterated 
until the resulting images converge as shown at 37, 

[0036] It should be noted that the use of the isotropic smoothing tn the chrominance bands together with any inter- 
polation method in the luminance band provides an improvement over current demosaicing algorithms with respect to 
the suppression of color artifacts at edges while preserving edge detail. The directional smoothing techniques discussed 
35 above provide improved performance in the luminance band, and hence, the preferred embodiment of the present 
invention utilizes both techniques. 

[0037] It should Denoted that the RGB to YIQ transformation and the smoothing operations are both linear operations. 
Accordingly, an equivalent algorithm can be generated that operates on the RGB image. Such an algorithm would 
eliminate the need to transform from RGB to YIQ and visa versa at each iteration,, and hence, impose a reduced 

40 computational load on the computing platform on which the method of the present invention is practiced. 

[0038] The method is preferably practiced on a genera! purpose computer. However it will bo obvious to those skilled 
in the art from the preceding discussion that it can be advantageously practiced on computational platforms having 
special purpose hardware for reducing the computational load imposed by linear transformations. 
[0039] The skilled person witf readily appreciate that this disclosure encompasses an image processing system hav- 

45 ing apparatus components operable to carry out each of the described method steps. The structure of such apparatus 
components will be immediately appamni to the skilled person from the teachings herein. 

[0040] The disclosures in United States patent application No. 09/303772, from which this application claims priority, 
and in the abstract accompanying this application are incorporated herein by reference. 

50 

Claims 



1, A method of operating a data processing system to generate a lull color image comprising a first two-dimensional 
array of vectors, each vector in said first two-dimensionaf array comprising first, second, and third intensity values, 
each intensity value representrng the intensity of a pixel rn said full cotor image in a corresponding spectral band 
at a location in said full color image determined by the location of said vector in said first two-dimensional array, 
said method generating said first two-dimensional array of vectors from a two-dimensional array of sealars. each 
scalar in said two-dimensional array of scalars corresponding to one of said vectors in said first two-dimensional 
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array of vectors, said scalar determining one of said first, second, or third intensity values at a corresponding 
location in said two-dimensional array ot vectors, said method determining the remaining ones of said first, second, 
and third intensity values, said method comprising the steps of: 

(a) assigning a value to each one of said. components of said vectors in said first two<Jimensiona^ array of 
vectors that is not determined by one of said scalers; 

(b) generating a luminance image and first and second chrominance images from said firs! two<iimensional 
array ot vectors, each pixel in said luminance image being determined by the intensity of light from a corre- 
sponding pixel in said f?rst two-dimenstonaf array and each pixel in said first and second chrominance images 
representing the color of said pixel in said first two-dimensional array; 

(c) filtering said first and second chrominance images with a isotropic low-pass spatial fiiter to generate first 
and second filtered chrominance images; 

(d) generating a second two-dimensional array of vectors from said lumrnance image and said first and second 
filtered chrominance images, each vector in said second two-d^mens tonal array comprising first, second, and 
third intensity values, each intensity value representing the intensity of a pixel in said full color image in a 
corresponding spectral band at a location in said full color image determined by the location of said vector in 
said first two-dimensional array: 

(e) setting a component of eaGh vector in said second two-dimensional array to said scalar value in said two- 
dimensional array of scaiars; 

(f) replacing said first two-dimensional array of vectors by said second two-dimensional array of vectors; and 

(g) repeating steps fb)-{f). 

A method as in Claim 1 comprising the step of filtering said luminance image with a low-pass spatial filter, said 
filter having an anisotropy that varies with location in said luminance image, said filtering being applied prior to 
generating satd second two-dimensional array of vectors. 

A method as rn Claim 2, wherein said step of filtering said luminance image comprises the steps of; 

decomposing said luminance image into a plurality of component images, each component image representing 
information in said luminance image at different levels oi scale; 

applying a low-pass spatial filter to each of said component images, said low-pass fiEter having an anisotropy 
that varies with location in said component image; and 

combining said filtered component images to regenerate a new luminance image, said new lumrnance image 
replacing said luminance image in steps (d)-(f). 

A method as in Claim 3, wherein said step of decomposing said luminance image comprises generating a Laplacian 
pyramid from sard luminance image 

A method as in Claim 2, wherein anisotrophy of said filter is determined by the gradient of said luminance image 
at said location in said luminance image. 

A method of interpolating an image comprising a two-dim ens renal array of pixel values ; each pixel value repre- 
senting an Intensity value for said image at a corresponding point in said image, said image including one or more 
unknown pixel values and a plurality of measured pixe^ values having known values; said method comprising the 
steps of; 

(a) assigning a value to each of said unknown pixe^ values: 

(b) filtering said image with a iow-pa$s filter having an anisotropy that varies with position in said image; 
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(c) resetting each measured pixel value to said known value: and 

fd) repeating steps (a)-(c). 

A method as in Claim 6 wherein said step of tittering said image comprises; 

decomposing said image \r\\o a plurality of component images, each component image representing informa- 
tion in saia image ai differed I levels of scale; 

applying a low-pass spatial filter to each of said component images, said low-pass filter having an anisotropy 
that vanes with location in said component image; and 

combining sad filtered component Images to regenerate a new image, said new image replacing said image 
in steps {c)-(d}, 

A melhod as in Claim 7, wherein said step of decomposing said image comprises generating a Laplacian pyramid 
from said image. 

A method as m Claim 7, wherein anisotrophy of said filter is determined by the gradient of said image at said 
location in said image. 
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(57) A method tor operating a data processing sys- 
tem to generate a full color image from a partially sam- 
pled version of the image. The full color image includes 
a first two-dimensional array of vectors having compo- 
nents representing the intensity of a pixel in the full color 
mnage in a corresponding spectral band at a location de- 
termined by the location of the vector in the first two- 
dimensional array. The method generates the first two- 
dimensional array from a two-dimensional array ot sca- 
lars. Each scalar determines one of the first, second, or 
third intensity values at a corresponding location in the 
two-dimensional array of vectors. The method deter- 
mines the remaining ones of the first, second, and third 
intensity values. The method starts by assigning a value 
to each one of the components of the vectors in the first 
two-dimensional array of vectors that is not determined 
by one of the scalane A lumtnance image and first and 
second chrominance images are then generated from 
the first two-dimensional array of vectors. The chromi- 
nance Images are filtered with an isotropic low-pass 
spatial filter to generate filtered chrominance images. 
The two-dknenskmai array of vectors Is then regenerat- 
ed from the \um\ nance image and the first and second 
filtered chrominance images. The scafars that are orig- 
inally given by the sensors are reset in the regenerated 
two-dimensional array of vectors. The decomposition, 
filtering, resetting, and regenerating steps are iterated 
to provide the final full-color image. In the preferred em- 
bodiment, the luminance image Is also filtered. Howev- 
er the filtering of trie luminance image utilizes a low- 



pass spatial fitter having an anisotropy that varies with 
location in the luminance rrnage. 
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